DECLARATION 




I , Toshiaki Sato, residing at 7 th Fl., Shuwa Kioicho Park 
Bldg., 3-6, Kioicho, Chiyoda-ku, Tokyo, Japan, hereby declare 
that I have a thorough knowledge of Japanese and English 
languages, and that the attached pages contains a correct 
translations into English of the application documents of 
Japanese Patent Application Nos. 7-196677 filed on August 
1, 1995, 7-205886 filed on August 11, 1995 and 7-214552 filed 
on August 23, 1995 in the name of CANON KABUSHIKI KAISHA. 

I further declare that all statements made herein of my 
own knowledge are true and that all statements made on 
information and belief are believed to be true; and further 
that these statement were made with the knowledge that willful 
false statements and the like so made, are punishable by fine 
or imprisonment, or both, under Section 1001 of Title 18 of 
the United States Code and that such willful false statements 
may jeopardize the validity of the application or any patent 
issuing thereon. 



Signed this 22nd day of March, 2006 




Toshiaki Sato 



Translation of Japanese Patent Application No. 7-196677 



[Type of Document (s) ] 
[Reference Number] 
[Filing Date] 
[Addressee] 

[International Patent 
Classification] 

[Title of Invention] 

[Number of Claim (s)] 

[ Inventor ( s ) ] 

[Address /Domicile] 

[Name] 

[ Inventor ( s ) ] 

[Address /Domicile] 

[Name] 

[ Inventor ( s ) J 

[Address/Domicile] 

[Name] 

[Applicant for Patent] 

[Identification Number] 

[Postal Code] 
[Address /Domicile] 

[Name] 

[Representative] 

[Agent] 

[Identification Number] 
[Postal Code] 



Application for patent 
3073006 

August 1, 1995 

Director-General of the Patent 
Office, Esq. 



H04N 5/225 

IMAGE REPRODUCING METHOD AND 
APPARATUS 

12 



c/o Canon Kabushiki Kaisha 
30-2, Shimomaruko 3-chome, 
Ohta-ku, Tokyo, Japan 

Yoshinobu SHIRAIWA 



c/o Canon Kabushiki Kaisha 
30-2, Shimomaruko 3-chome, 
Ohta-ku, Tokyo, Japan 

Eiichiro IKEDA 



c/o Canon Kabushiki Kaisha 
30-2, Shimomaruko 3-chome, 
Ohta-ku , Tokyo , Japan 

Yoshiro UDAGAWA 

000001007 
146 

30-2, Shimomaruko 3-chome, 
Ohta-ku, Tokyo, -Japan 

Canon Kabushiki Kaisha 

Hajime MITARAI 

100066061 
105 



[Address /Domicile] 

[Patent Attorney] 

[Name] 

[Telephone] 

[Selected Agent] 

[Identification Number] 
[Postal Code] 
[Address /Domicile] 

[Patent Attorney] 

[Name] 

[Telephone] 

[Detail of Fee(s) ] 

[Method of Payment] 

[Register Number of 
Prepayment] 

[Amount of Payment] 

[List of Attached Documents] 
[Classification] 
[Classification] 
[Classification] 

[Number of General Power of 

Attorney] 

[Proof Required?] 



Nihonseimei Shinbashi Bldg. 3F 
18-16, Shinbashi 1-chome, 
Minato-ku, Tokyo, Japan 

Hiroyuki TANBA 
03-3503-2821 

100094754 
105 

Nihonseimei Shinbashi Bldg. 3F 
18-16, Shinbashi 1-chome, 
Minato-ku, Tokyo , Japan 

Tadao NOGUCHI . 
03-3503-2821 



Prepayment 

011707 
21000 

Specification 1 
Drawing (s) 1 
Abstract 1 

9004560 
Yes 



7-196677 



(Type of the Document] Specification 

[Title of the Invention] IMAGE REPRODUCING METHOD AND 

APPARATUS 

[What Is Claimed Is:] 
5 [Claim 1] An image reproducing method 

characterized by converting image sensing data obtained 
by image sensing means into a visualizable image signal 
by using an image reproduction parameter, and 
comprising forming a group of image sensing data 

10 consisting of a plurality of image sensing data meeting 
a predetermined condition, determining the image 
reproduction parameter on the basis of the image 
sensing data group, and converting the image sensing 
data belonging to the image sensing data group into 

15 image data by using the image reproduction parameter. 

[Claim 2] The image reproducing method according 
to. claim 1, characterized in that the group of image 
sensing data consisting of a plurality of image sensing 
data meeting the predetermined condition are image 

20 sensing data whose color temperature data fall within a 
previously designated color temperature data range, the 
color temperature data being obtained by analyzing the 
image sensing data. 

t Claim 3] The image reproducing method according 

25 to claim 1, characterized in that the group of image 

sensing data consisting of a plurality of image sensing 
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data meeting the predetermined condition are image 
sensing data selected by a user from a group of images 
which are displayed through simple reproduction of the 
imag e sensing dat a . 
5 [Claim 4] The image reproducing method according 

to claim 1, characterized in that the group of image 
sensing data consisting of a plurality of image sensing 
data meeting the predetermined condition are image 
sensing data in which one of pieces of information 

10 appended to the image sensing data, such as a 

photographing time, an EV value, an aperture value, an 
object distance, and a shutter speed, or a combination 
of these pieces of information falls within a 
previously numerically designated range. 

15 [Claim 5] The image reproducing method according 

to claim 1, characterized in that the group of image 
sensing data consisting of a plurality of image sensing 
data meeting the predetermined condition are image 
sensing data in which one of pieces of information 

20 appended to the image sensing data, such as a 
photographing mode, a photographing place, the 
use/nonuse of an electronic flash, and the use/nonuse 
of an optical low-pass filter, or a combination of 
these pieces of information meets a previously 

25 designated condition. 

[Claim 6] The image reproducing method according 
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to claim 1, characterized in that the group of image 
sensing data consisting of a plurality of image sensing 
data meeting the predetermined condition are image 
sensing data having information appended thereto which 
5 indicates that the image sensing data are adapted to 
form a synthetic image. 

[Claim 7] An image reproducing apparatus 
characterized in that said apparatus converts image 
sensing data obtained by image sensing means into a 

10 visualizable image signal by using an image 

reproduction parameter, and comprises image sensing 
data memory means for storing obtained image sensing 
data, image sensing data selection designating means 
for designating a condition for image sensing data 

15 selection, image sensing data selecting means for 

selecting image sensing data from said image sensing 
data memory means, image sensing data holding means for 
holding the image sensing data selected by said image 
sensing data selecting means, image reproduction 

20 parameter determining means for determining an image 
reproduction parameter by using the selected image 
sensing data held by said image sensing data holding 
means, and image reproduction processing means for 
performing image reproduction on the basis of the image 

25 sensing data held by said image sensing data holding 
means by using the image reproduction parameter 
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determined by said image reproduction parameter 
determining means, 

wherein a group of image sensing data consisting 
of a plurality of image sensing data meeting a 
5 predetermined condition are formed, the image 

reproduction parameter is determined on the basis of 
the group of image sensing data, and the image sensing 
data belonging to the image sensing data group are 
converted into image data by using the image 

10 reproduction parameter. 

[Claim 8] The image reproducing apparatus 
according to claim 7, characterized in that the group 
of image sensing data consisting of a plurality of 
image sensing data meeting the predetermined condition 

15 are image sensing data whose color temperature data 
fall within a previously designated color temperature 
data range, the color temperature data being obtained 
by analyzing the image sensing data. 

[Claim 9] The image reproducing apparatus 

20 according to claim 7, characterized in that the group 
of image sensing data consisting of a plurality of 
image sensing data meeting the predetermined condition 
are image sensing data selected by a user from a group 
of images which are displayed through simple 

25 reproduction of the image sensing data. 

[Claim 10] The image reproducing apparatus 
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according to claim 7 , characterized in that the group 
of image sensing data consisting of a plurality of 
image sensing data meeting the predetermined condition 
are image sensing data in which one of pieces of 
5 information appended to the image sensing data, such as 
a photographing time, an EV value, an aperture value, 
an object distance, and a shutter speed, or a 
combination of these pieces of information falls within 
a previously numerically designated range. 

10 [Claim 11] The image reproducing apparatus 

according to claim 7, characterized in that the group 
of image sensing data consisting of a plurality of 
image sensing data meeting the predetermined condition 
are image sensing data in which one of pieces of 

15 information appended to the image sensing data, such as 
a photographing mode, a photographing place, the 
use/nonuse of an electronic flash, and the use/nonuse 
of an optical low-pass filter, or a combination of 
these pieces of information meets a previously 

20 designated condition. 

[Claim 12] The image reproducing apparatus 
according to claim 7, characterized in that the group 
of image sensing data consisting of a plurality of 
image sensing data meeting the predetermined condition 

25 are image sensing data having Information appended 

thereto which indicates that the image sensing data are 
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adapted to form a synthetic image. 
[Detailed Description of the Invention] 
[0001] 

[Technical Field of the Invention] 
5 The present relates to the technical field of an 

image reproducing method and apparatus for converting 
image data of an image sensing signal obtained by an 
image sensing device such as an image sensing tube or a 
CCD into a visualizable image signal, e.g., an NTSC-RGB 
10 signal or the like. 
[0002] 

[Prior Art] 

In a conventional image reproducing apparatus such 
as a television camera using an image sensing device 

15 such as a CCD, some image reproduction parameters are 
generally determined from image sensing data during 
image reproduction processing in which image data are 
obtained from the image sensing data, in order to 
constantly obtain images which apparently give the same 

20 impression or to obtain as faithful reproduced images 
as possible regardless of deterioration with time of 
the image sensing device or a color filter and changes 
in an illuminating light source. Examples of the image 
reproduction parameters are a color temperature and a 

25 reproduction luminance level. The image reproduction 

parameters are used to correct the color temperature or 
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set the reproduction luminance level. More 
specifically, the correction of the color temperature 
is to adjust a so-called white balance so that an 
object which is supposed to look white looks white, 
5 Generally, this color temperature correction is 

performed on the basis of image sensing data. That is, 
image data which is supposed to look white is extracted 
from image sensing data, and a white balance 
coefficient as one image reproduction parameter is 

10 determined on the basis of the extracted data. In the 
white balance adjustment, a plurality of color 
component signals constituting an output image signal 
from an image sensing device are, respectively, 
provided with gains and corrected using the white 

15 balance coefficient. Consequently, the signal levels 
of the color components constituting the image signal 
of the object which is supposed to look white are so 
adjusted as to be equal to each other. 
[0003] 

20 The setting of the reproduction luminance level is 

done by calculating a luminance distribution from image 
sensing data and setting an optimum reproduction 
luminance level (range). The parameter is adjusted 
such that a reproduced image is obtained within this 

25 range, and the image is reproduced. 
[0004] 
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[Problems That the Invention Is to Solve] 
The method of adjusting the hue of an image by 
adjusting the white balance as described above is 
effective when a sufficient number of objects which are 
5 supposed to look white exist in an image signal 

obtained from an image sensing device. However, no 
such object which is supposed to look white exists in 
an image signal or only a very few such objects exist 
in an image signal in specific instances. In these 

10 instances, therefore, it is in principle impossible to 
adjust the hue by adjusting the white balance. In such 
instances, the general approach is to average the image 
signal containing the entire image sensing data of one 
image plane for each color component and adjust the 

15 white balance by using the average, in order to avoid 
the problem. However, a color indicated by the 
obtained average is not necessarily white (the color of 
a light source), and so the white balance cannot be 
accurately adjusted. 

20 [0005] 

Thus, in the conventional image reproducing 
apparatus, the image reproduction parameter cannot be 
accurately set if it is determined from image sensing 
data when image reproduction processing is performed in 
25 order to obtain an optimum reproduced image, 
[0006] 



- 8 - 



7-196677 



Also, in the setting of the reproduction luminance 
level, if the reproduction luminance parameter is 
determined for each image sensing data, the correlation 
between the luminances of a plurality of image sensing 
5 data is lost when image comparison or image synthesis 
processing is performed. This makes the comparison of 
reproduced images difficult, or the connection of 
luminances becomes unnatural when the reproduced images 
are synthesized, 
10 [0007] 

For example, the above inconveniences are 
significant when an object which is to be originally, 
desirably photographed as one image plane is 
divisionally photographed and image sensing data 
15 corresponding to the plurality of image planes are 
obtained. 
[0008] 

That is, when a reproduced image is obtained by 
using an image reproduction parameter determined from 

20 image information, an independent image reproduction 

parameter is obtained for each image sensing data. For 
this reason, it is impossible to obtain a reproduced 
image which is used when information between a 
plurality of images is extracted by comparing and 

25 analyzing the images, such as when physical property 
information is obtained from luminance information. 
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Also, if the reflectance of an object spatially, 
gradually changes, individual image sensing data 
obtained by divisionally photographing the object have 
different luminance levels (ranges). If images are 
reproduced by independently optimizing these image 
sensing data, the correlation between luminances 
originally corresponding to the respective image 
sensing areas is lost in the reproduced images. 
Accordingly, if one image is formed by synthesizing 
these images taken in the respective image sensing 
areas, an unnatural synthetic image in which the 
correlation between luminances is lost results. 
[0009] 

The present invention has been made to solve the 
above problems, and has as its object to provide an 
image reproducing method and apparatus capable of 
performing desired accurate image reproduction. 
[0010] 

[Means of Solving the Problems] 
In order to achieve the above object, there are 
provided an image reproducing method and an image 
reproducing apparatus in accordance with the present 
invention as follows: 

(1) The present invention provides an image 
reproducing method characterized by converting image 
sensing data obtained by image sensing means into a 
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visualizable image signal by using an image 
reproduction parameter, and comprising forming a group 
of image sensing data consisting of a plurality of 
image sensing data meeting a predetermined condition, 
determining the image reproduction parameter on the 
basis of the image sensing data group, and converting 
the image sensing data belonging to the image sensing 
data group, into image data by using the image 
reproduction parameter. 
[0011] 

(2) The present invention provides the image 
reproducing method according to (1), characterized in 
that the group of image sensing data consisting of a 
plurality of image sensing data meeting the 
predetermined condition are image sensing data whose 
color temperature data fall within a previously 
designated color temperature data range, the color 
temperature data being obtained by analyzing the image 
sensing data. 

[0012] 

(3) The present invention provides the image 
reproducing method according to (1), characterized in 
that the group of image sensing data consisting of a 
plurality of image sensing data meeting the 
predetermined condition are image sensing data selected 
by a user from a group of images which are displayed 
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through simple reproduction of the image sensing data. 
[0013] 

(4) The present invention provides the image 
reproducing method according to (1), characterized in 
that the group of image sensing data consisting of a 
plurality of image sensing data meeting the 
predetermined condition are image sensing data in which 
one of pieces of information appended to the image 
sensing data, such as a photographing time, an EV value, 
an aperture value, an object distance, and a shutter 
speed, or a combination of these pieces of information 
falls within a previously numerically designated range. 
[0014] 

(5) The present invention provides the image 
reproducing method according to (1), characterized in 
that the group of image sensing data consisting of a 
plurality of image sensing data meeting the 
predetermined condition are image sensing data in which 
one of pieces of information appended to the image 
sensing data, such as a photographing mode, a 
photographing place, the use/nonuse of an electronic 
flash, and the use/nonuse of an optical low-pass filter, 
or a combination of these pieces of information meets a 
previously designated condition. 

[0015] 

(6) The present invention provides an image 
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reproducing method according to (1), characterized in 
that the group of image sensing data consisting of a 
plurality of image sensing data meeting the 
predetermined condition are image sensing data having 
5 information appended thereto which indicates that the 
image sensing data are adapted to form a synthetic 
image. 
[0016] 

(7) The present invention provides an image 

10 reproducing apparatus characterized in that the 

apparatus converts image sensing data obtained by image 
sensing means into a visualizable image signal by using 
an image reproduction parameter, and comprises image 
sensing data memory means for storing obtained image 

15 sensing data, image sensing data selection designating 
means for designating a condition for image sensing 
data selection, image sensing data selecting means for 
selecting image sensing data from the image sensing 
data memory means, image sensing data holding means for 

20 holding the image sensing data selected by the image 
sensing data selecting means, image reproduction 
parameter determining means for determining an image 
reproduction parameter by using the selected image 
sensing data held by the image sensing data holding 

25 means, and image reproduction processing means for 

performing image reproduction on the basis of the image 
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sensing data held by the Image sensing data holding 
means by using the image reproduction parameter 
determined by the image reproduction parameter 
determining means , 

wherein a group of image sensing data consisting 
of a plurality of image sensing data meeting a 
predetermined condition are formed, the image 
reproduction parameter is determined on the basis of 
the group of image sensing data, and the image sensing 
data belonging to the image sensing data group are 
converted into image data by using the image 
reproduction parameter. 
[0017] 

(8) The present invention provides the image 
reproducing apparatus according to (7), characterized 
in that the group of image sensing data consisting of a 
plurality of image sensing data meeting the 
predetermined condition are image sensing data whose 
color temperature data fall within a previously 
designated color temperature data range, the color 
temperature data being obtained by analyzing the image 
sensing data. 

[0018] 

(9) The present invention provides the image 
reproducing apparatus according to ( 7 ) , characterized 
in that the group of image sensing data consisting of a 
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plurality of image sensing data meeting the 
predetermined condition are image sensing data selected 
by a user from a group of images which are displayed 
through simple reproduction of the image sensing data, 
5 10019] 

(10) The present invention provides the image 
reproducing apparatus according to (7), characterized 
in that the group of image sensing data consisting of a 
plurality of image sensing data meeting the 

10 predetermined condition are image sensing data in which 
one of pieces of information appended to the image 
sensing data, such as a photographing time, an EV value, 
an aperture value, an object distance, and a shutter 
speed, or a combination of these pieces of information 

15 falls within a previously numerically designated range, 
[0020] 

(11) The present invention provides the image 
reproducing apparatus according to (7), characterized 
in that the group of image sensing data consisting of a 

20 plurality of image sensing data meeting the 

predetermined condition are image sensing data in which 
one of pieces of information appended to the image 
sensing data, such as a photographing mode, a 
photographing place, the use/nonuse of an electronic 

25 flash, and the use/nonuse of an optical low-pass filter, 
or a combination of these pieces of information meets a 
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previously designated condition. 
[0021] 

(12) The present invention provides the image 
reproducing apparatus according to (7), characterized 
in that the group of image sensing data consisting of a 
plurality of image sensing data meeting the 
predetermined condition are Image sensing data having 
information appended thereto which indicates that the 
image sensing data are adapted to form a synthetic 
image. 
[0022] 

The present invention achieves the above-mentioned 
object with the respective arrangements as described 
above . 
[0023] 

[Embodiments] 
In order to achieve the above object, according to 
the present invention, an image reproducing method and 
apparatus for converting image sensing data obtained by 
image sensing means into a visualizable image signal by 
using an image reproduction parameter are characterized 
in that the image reproduction parameter is determined 
from a group of image sensing data consisting of a 
plurality of image sensing data meeting a predetermined 
condition and the image sensing data belonging to the 
image sensing data group are converted into image data 
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by using the image reproduction parameter. 
[0024] 

The image reproducing method and apparatus are 
further characterized in that the group of image 
5 sensing data consisting of a plurality of image sensing 
data meeting the predetermined condition are image 
sensing data whose color temperature data fall within a 
previously designated color temperature data range, the 
color temperature data being obtained by analyzing the 

10 image sensing data. 

The image reproducing method and apparatus are 
further characterized in that the group of image 
sensing data consisting of a plurality of image sensing 
data meeting the predetermined condition are image 

15 sensing data selected by a user under a condition that 
the compositions of an object displayed through simple 
reproduction of the image sensing data are similar to 
each other. 
[0025] 

20 The image reproducing method and apparatus are 

further characterized in that the group of image 
sensing data consisting of a plurality of image sensing 
data meeting the predetermined condition are image 
sensing data in which one of pieces of information 

25 appended to the image sensing data, such as a 

photographing time, an EV value, an aperture value, an 
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object distance, and a shutter speed, or a combination 
of these pieces of information falls within a 
previously numerically designated range. 
[0026] 

The image reproducing method and apparatus are 
further characterized in that the group of image 
sensing data consisting of a plurality of image sensing 
data meeting the predetermined condition are image 
sensing data in which one of pieces of information 
appended to the image sensing data, such as a 
photographing mode, a photographing place, the 
use/nonuse of an electronic flash, and the use/nonuse 
of an optical low-pass filter, or a combination of 
these pieces of information meets a previously 
designated condition. 
[0027] 

Furthermore, the image reproducing method and 
apparatus are characterized in that the group of image 
sensing data consisting of a plurality of image sensing 
data meeting the predetermined condition are a 
plurality of image sensing data having information as 
image sensing data adapted to form a synthetic image or 
a plurality of image sensing data selected as image 
sensing data adapted to form a synthetic image. 
[0028] 

According to the above -described image reproducing 
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method and apparatus, image sensing data of a plurality 
of image planes associated to each other are processed 
as a group of image sensing data and an optimum image 
reproduction parameter is determined by using the image 
sensing data. The image sensing data constituting the 
image sensing data group are converted into image data 
by using the image reproduction parameter. 
[0029] 

[Embodiment] 

An embodiment of the present invention will be 
described below with reference to the accompanying 
drawings . 
[0030] 

Fig. 1 is a block diagram showing an embodiment of 
an image reproducing apparatus according to the present 
invention. 
[0031] 

Reference numeral 10 denotes a color image sensing 
unit, such as a digital camera, which senses the image 
of an object and outputs the image data of the object 
to an image sensing data memory 20 as two-dimensional 
digital image sensing data (hereinafter referred to 
simply as "image sensing data"). The image sensing 
data memory 20 stores and holds the image sensing data 
supplied from the color image sensing unit 10. 
[0032] 
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A color image reproduction processor 30 performs 
predetermined image reproduction processing for the 
image sensing data stored and held in the image sensing 
data memory 20. For example, the color image 
reproduction processor 30 converts the image sensing 
data into digital NTS C- RGB data and outputs the digital 
data to a color image reproducing display 40 and a 
reproduced image data memory 50. 
[0033] 

The color image reproducing display 40 includes a 
color video card and a monitor, for example. The color 
image reproducing display 40 receives an output color 
image signal from the color image reproduction 
processor 30 or reads out a color image signal from the 
reproduced image data memory 50 and displays the signal 
as a color image on the monitor. 
[0034] 

The reproduced image data memory 50 stores the 
image data reproduced by the color image reproduction 
processor 30. The image data stored in the reproduced 
image data memory 50 is supplied to and displayed by 
the color image reproducing display 40 as needed. 
[0035] 

More specifically, the color image reproduction 
processor 30 comprises an image sensing data holding 
mechanism 31 for holding the digital image sensing data 
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from the image sensing data memory 20, an image sensing 
data selecting mechanism 33 for selecting image sensing 
data from the image sensing data memory 20 and 
outputting the selected data to the image sensing data 
5 holding mechanism 31 f an image sensing data selection 
designating unit (designating means) 34 for designating 
the condition or the like by which the image sensing 
data selecting* unit 33 selects image sensing data, an 
image reproduction processing unit 35 for reproducing 

10 an image of the image sensing data held by the image 
sensing data holding unit 31 by using an image 
reproduction parameter determined by an image 
reproduction parameter determining unit 32 adapted to 
determine the image reproduction parameter by using the 

15 image sensing data held by the image sensing data 

holding unit 31, and a control unit 36 for controlling 
these units of the color image reproduction processor 
30. 

[0036] 

20 Examples of the image reproduction parameter 

determined by the image reproduction parameter 
determining unit 32 are parameters indicating a color 
temperature, a white balance coefficient, a color 
component gain, white point information, black point 

25 information, a gamma coefficient, a gradation 
characteristic, a gradation conversion curve, a 
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gradation conversion lookup table, a knee point, a 
dynamic range, a color gamut, light source inf ormation, 
a color coordinate conversion matrix coefficient, a 
spatial frequency characteristic, a black balance 
coefficient, an S/N ratio, an auto -correlation 
coefficient, a Wiener spectrum, an intensity (density) 
distribution, and a luminance distribution, and 
parameters obtained directly or indirectly from these 
pieces of information. 
[0037] 

The image sensing data selection designating unit 
34 can designate the condition of selection in order 
that of a plurality of image sensing data stored in the 
image sensing data memory 20, image sensing data 
meeting a predetermined condition be processed as a 
group of image sensing data. The designated condition 
is given, e.g., as a condition related to color 
temperature information or the composition of an object 
obtained from image sensing data, or as a predetermined 
condition pertaining to the photographing time, the 
photographing mode, the photographing place, the EV 
value, the aperture value, the object distance, the 
shutter speed, the use/nonuse of an electronic flash, 
or the use/nonuse of an optical low-pass filter, each 
of which is information appended to image sensing data. 
[0038] 
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Fig.t2 is a flow chart showing the procedure of 
image processing done by the image reproducing 
apparatus of this embodiment. This program is executed 
by the color image reproduction processor 30. Assume 
5 that a plurality of image sensing data obtained by the 
color image sensing unit 10 are already stored in the 
image sensing data memory 20 . 
[0039] 

When the flow chart shown in Fig.t2 starts, the 

10 control unit 36 for performing the processing first 
checks the contents of image sensing data selection 
designated (in step SI) by the image sensing data 
selection designating unit 34 and sends the contents of 
the designated selection to the image sensing data 

15 selecting unit 33. Then, on the basis of the contents 
of the selection designated by the image sensing data 
selection designating unit 34, the image sensing data 
selecting unit 33 selects image sensing data meeting 
the contents of the designated selection from the image 

20 sensing data stored in the image sensing data memory 20 
and outputs the selected data to the image sensing data 
holding unit 31 <S2). If the image sensing data 
meeting the contents of the designated selection is 
data of less than one image plane, this program is not 

25 executed. If the selected image sensing data is data 

of one image plane, the image sensing data is converted 
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into image data by ordinary image reproduction 
processing. If there is no image sensing data to be 
selected, information indicating this is output, and 
the processing is terminated, 
[0040] 

The image sensing data holding unit 31 temporarily 
holds the image sensing data selectively output from 
the image sensing data memory 20 (S3). 
[0041] 

The image reproduction parameter determining unit 
32 analyzes the image sensing data temporarily held by 
the image sensing data holding unit 31 using the image 
sensing data, obtains, e.g., light source information 
(e.g., the color temperature or the chromaticity of the 
light source) when the image sensing data are obtained, 
which is necessary in white balance adjustment, and 
determines an image reproduction parameter on the basis 
of the light source information (S4). That is, by 
using the plurality of image sensing data selected and 
held as a group of image sensing data, the image 
reproduction parameter determining unit 32 determines 
an image reproduction parameter and sends the parameter 
to the image reproduction processing mechanism 35. 
[0042] 

By using the image reproduction parameter thus 
determined, the image reproduction processing unit 35 
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converts the image sensing data constituting of the 
image sensing data group held by the image sensing data 
holding unit 31 into image data (S5). 
[0043] 

5 The conditions under which image sensing data are 

selected from a plurality of image sensing data so as 
to be processed as one group of image sensing data will 
be described below. 
[0044] 

10 Information obtained from the analysis of image 

sensing data itself includes color temperature 
information and the composition of an object. It is 
sometimes necessary to process a plurality of image 
sensing data whose color temperatures are within a 

15 certain fixed range as a group of correlated image 
sensing data. Especially when the number of samples 
used to estimate color temperatures is small, the 
estimation accuracy can be increased by again 
estimating the color temperatures by using a plurality 

20 of image sensing data whose estimated color 

temperatures are within a certain fixed range. 
Consequently, image reproduction can be performed with 
a high accuracy. 
[0045] 

25 If the compositions of objects are the same or 

very similar, there is a high possibility that the 
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objects were photographed under the same photographing 
condition . Accordingly, the image reproduction 
parameters of the images are the same or can be 
processed as the same parameter within a certain fixed 
range of accuracy. If this is the case, an image 
reproduction parameter can be accurately obtained when 
it is obtained from a plurality of image sensing data. 
It is in many instances useful to process image sensing 
data having the same object composition or similar 
object compositions as a group of image sensing data. 
[0046] 

Examples of information useful to find the 
correlation between image sensing data are, e.g., the 
photographing time (date), the photographing mode, the 
photographing place, the EV value, the shutter speed, 
the aperture value, the object distance, the use/nonuse 
of an electronic flash, and the use/nonuse of an 
optical low-pass filter. If these pieces of 
information are given as information appended to image 
sensing data, a group of image sensing data can be 
generated by analyzing these pieces of information. 
[0047] 

For example, if the photographing times of image 
sensing data are within a certain fixed range, it is 
obviously effective to process these data as a group of 
image sensing data. It is also clearly effective to 
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process a series of image sensing data whose imaging 
mode is a continuous shooting mode as a group of image 
sensing data. As for information pertaining to the 
use/nonuse of an electronic flash, information of the 
5 photographing light source largely depends upon the 
characteristics of the flash light. Therefore, it is 
useful to simultaneously process a plurality of image 
sensing data, obtained by using an electronic flash, as 
a group of image sensing data. 
10 [0048] 

Pieces of information such as the EV value, the 
shutter speed, the aperture value, and the object 
distance extremely largely depend upon the 
photographing conditions and function as strong 

15 restricting conditions when a reproduction parameter is 
determined. Therefore, image sensing data having these 
values within a predetermined range are processed as 
one image sensing data group. This is very efficient 
when image reproduction is performed. 

20 [0049] 

In information of the photographing place, the 
characteristics of object remain unchanged. Therefore, 
this information is very effective in accurately 
reproducing an object. Additionally, when this 
25 information is used in combination with information of, 
e.g., the photographing time (date), information on a 
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change in the photographing conditions can also be 
obtained. This obviously increases the usefulness of 
the information of the photographing place, 
[0050] 

In the above example, the condition for each 
individual information is primarily described. However, 
it is evident that the effectiveness is sometimes 
greatly improved by combining the above conditions. 
[0051] 

As shown in Fig. 3, for example, as the condition 
of processing, a plurality of image sensing data having 
a spatial continuity are sometimes designated to obtain 
a synthetic image. If this is the case, it is 
necessary to hold the spatial continuity in the 
synthetic image. For this purpose, it is very 
effective to perform image reproduction processing by 
processing the above-described plurality of image 
sensing data as one group of image sensing data. 
[0052] 

Also, as illustrated in Fig.t4, if it can be 
determined from the photographing place information or 
composition information obtained by analyzing image 
sensing data that the image sensing data have a time 
continuity and are obtained from objects which are 
nearly the same, a reproduced image maintaining the 
time continuity can be obtained by processing these 



- 28 - 



7-196677 



image sensing data as one group of image sensing data 
using the image reproducing method and apparatus 
according to the present invention, 
[0053] 

5 If these image sensing data are obtained outdoors, 

for example, an image can be reproduced while a 
continuous change with a change in the photographing 
time, i.e., the condition of sunshine, is held. More 
specifically, a reproduced image having a time 

10 continuity can be obtained by continuously changing an 
image reproduction parameter of image sensing data of 
an image sensing data group consisting of a plurality 
of image sensing data obtained from objects which are 
almost the same. This allows reproduction of an image 

15 while correcting disturbance occurring in some image 
sensing data due to a sudden change in the conditions. 
As a consequence, a temporally natural image can be 
obtained. 
[0054] 

20 Also, as shown in Fig. 5, image sensing data 

having both a time continuity and a continuity in the 
photographing order, i.e., image sensing data obtained 
when the photographing mode is a continuous shooting 
mode, or image sensing data also having a spatial 

25 continuity, are processed as one group of image sensing 
data. Consequently, an extraneous condition occurring 
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in each individual image sensing data can be eliminated, 
and this allows reproduction of an image keeping the 
continuity. 
[0055] 

A method of grouping image sensing data, i.e., a 
method of obtaining a group of image sensing data, by 
using color temperature information obtained from the 
analysis of the image sensing data itself will be 
described in detail below. Fig.t6 is a flow chart 
showing the image sensing data grouping processing. 
[0056] 

The conditions given by the image sensing data 
selection designating unit 34 are a correlated color 
temperature ( A(K) ) and its selecting range color 
temperature ( ±dA(K) ) . The correlated color temperature 
range is sometimes directly input by the user as a 
numerical value. It is also possible to input 
information related to the correlated color temperature, 
e.g., words such as "daytime", "evening", "sunset glow", 
"cloud", and "incandescent lamp", which the user can 
image in relation to the color temperature. Thus, 
color temperature ranges corresponding to these words 
can be previously defined by a system and stored in a 
memory. A color temperature range corresponding to the 
information input by the user can be read out and used, 
or can be obtained by calculations (step Sll in Fig. 6). 



- 30 - 



7-196677 



It is also possible to use adjectives such as "red" and 
"pale" from which the color temperatures can be 
imagined or practical values of chromaticity or a color 
difference. 
5 [0057] 

When the above conditions are designated, the 
correlated color temperature as the intrinsic 
information of image sensing data can be designated by 
using, e.g., a chromaticity diagram shown in Fig. 7. 
10 [0058] 

[Equation 1] 

Fig. 7 is a graph shown as Appended Figure 1 in 
JIS Z 8725 "Methods for Determining Distribution 
Temperature and Color Temperature or Correlated Color 
15 Temperature of Light Source". In Fig. 7, T cp " a is a 

reciprocal correlated color temperature T^" 1 (MK 1 ) and 
has the following relationship with a correlated color 
temperature T^: 

T^K) = lOVT^MK" 1 ) 
20 [0059] 

The above condition is supplied to the image 
sensing data selecting unit 33 and used as the 
condition of grouping. 
[0060] 

25 The image sensing data selecting unit 33 

sequentially loads the image sensing data stored in the 
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image data memory 20 (S12), analyzes the image sensing 
data and extracts information pertaining to the color 
temperature (S13), and checks whether the color 
temperature information satisfies the previously 
5 designated grouping condition (S14) . If the color 
temperature information satisfies the condition, the 
image sensing data selecting unit 33 supplies the 
corresponding image sensing data to the image sensing 
data holding unit 31 (S15). 

10 [0061] 

[Equation 2] 

Generally, the color temperature information 
obtained by analyzing the image sensing data is not a 
correlated color temperature itself but another signal 

15 having a one-to-one correspondence with the correlated 
color temperature. For example, when an image sensing 
device from which an R,G,B signal is obtained is used, 
it is possible to use, as the signal pertaining to the 
color temperature information, an (R/G,B/G) signal 

20 obtained by averaging image sensing data of one image 
plane or an (R/G,B/G) signal obtained as an average 
value of regions (which meet the condition that, e.g., 
their chromaticity values are within the range which 
defines the correlated color temperature in Fig. 7) 

25 sampled from image sensing data and corresponding to 
white. An example of the (R/G,B/G) signal is shown in 
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Fig. 8. Since the (R/G,B/G) signal depends upon the 
sensitivity characteristic of the image sensing device 
used with respect to each color component, it is 
necessary to obtain the correspondence to the 
correlated color temperature by taking into account the 
characteristics of an image sensing device used as a 
system. Since the characteristics of the (R/G,B/G) 
signals and the correlated color temperatures whose 
correspondence is thus obtained are in a one-to-one 
correspondence with each other, the relationship 
between the correlated color temperature (A(K) ) and its 
selecting range color temperature (±dA(K) ) which are 
the grouping conditions previously given can also be 
defined as follows by using the (R/G, B/G) signal. That 
is, the range of the correlated color temperatures of 
image sensing data to be grouped can be defined as 
A±dA(k>, or ((R/G) 0 ±d(R/G) 0 ,(B/G) 0 ±d(R/G) 0 ). Accordingly, 
to directly check by using the (R/G,B/G) signal whether 
the grouping conditions are met, it is only necessary 
to check whether the color temperature information (R/G, 
B/G) obtained by analyzing the image sensing data is 
within the range of ( (R/G) 0 ±d(R/G) 0 , (B/G) 0 ±d(R/G) 0 ) . To 
make this determination by using the correlated color 
temperature, a correlated color temperature 
corresponding to the (R/G, B/G) signal directly obtained 
from the image sensing data is obtained, and whether 



- 33 - 



7-196677 



this correlated color temperature is within the range 
of A±dA(K) as the grouping condition is checked. When 
a condition related to the color temperature 
information is given as the grouping condition, the 
above processing is sequentially executed for the image 
sensing data stored in the image sensing data memory 20 
It is determined that image sensing data meeting the 
condition are image sensing data constituting the group 
The image sensing data thus found to constitute the 
group are supplied to the image sensing data holding 
unit 31 (SI 5) and held in it. 
[0062] 

A method of grouping image sensing data when the 
composition of an object is designated as the condition 
of grouping will be described below. Fig.t9 is a flow 
chart showing the processing. 
[0063] 

If the composition is designated as the grouping 
condition, the image sensing data selecting unit 33 
loads the image sensing data stored in the image 
sensing data memory 20 (steps S21 and S22 in Fig. 9). 
Simplified reproduction is performed to the extent to 
which the composition is confirmed (S23), thereby 
obtaining simple reproduced image data. The simplified 
reproduction can be reproduction processing to the 
extent to which the composition can be confirmed. For 
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example, pixels are thinned from image sensing data and 
luminance data is reproduced while the data amount is 
thus decreased. 
[0064] 

Subsequently, the determination of grouping is 
performed by using the simple reproduced image data 
(S29). However, it is generally efficient to 
subjectively determine, e.g., the similarity between 
images. Therefore, this subjective determination 
method is used in this embodiment of the present 
invention. That is, if an instruction from the user 
can be expected in the determination of grouping, a 
series of simple reproduced image data are displayed on 
a monitor (S25). A means for allowing the user to 
select images to be grouped is arranged, and image 
sensing data corresponding to the images selected by 
the user are sequentially supplied to the image sensing 
data holding unit 31 and held in it (S26 and S30). 
[0065] 

To automatically determine grouping, for example, 
the following procedure is performed. That is, edge 
image data is formed by extracting an edge component 
from simple reproduced image data, and the sameness or 
the similarity of the spatial relationship between 
edges is detected by applying an image processing 
technology (S27 and S28). The spatial correlation 
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obtained by analyzing the sameness or the similarity is 
checked, and it is determined that images having a 
strong correlation are images meeting the grouping 
condition, and then image sensing data corresponding to 
the images are supplied to the image sensing data 
holding unit 31 and held in it (S29 and S30). 

A method of grouping image sensing data by using 
information appended to the image sensing data will be 
described below. Fig. 10 is a flow chart showing an 
example of the grouping processing. 
[0066] 

Assume that information appended to image sensing 
data and the condition of the information are given as 
the condition of grouping to the image sensing data 
selection designating unit 34. The image sensing data 
selecting unit 33 checks whether the information 
corresponding to the conditions is appended to the 
image sensing data stored in the image sensing data 
memory 20, and sequentially loads image sensing data to 
which the information is appended (steps S31 and S32 in 
Fig. 10). The image sensing data selecting unit 33 
then checks whether the appended information meets the 
designated condition of selection for grouping (S33 and 
S34). 
[0067] 

If information such as the photographing time, the 
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EV value, the aperture value, the object distance, or 
the shutter speed whose range can be numerically set is 
used, the grouping selection condition is whether the 
value of the appended information falls within the 
numerically indicated range. If information such as 
the image sensing mode, the photographing place, the 
use/nonuse of an electronic flash, or the use/nonuse of 
an optical low-pass filter which is difficult to 
numerically indicate is used, the agreement between the 
names of the places, the names of the modes, or the 
presence/absence is the condition. If the grouping 
information (the same mark is given to image sensing 
data obtained in the continuous shooting mode) 
depending on the image sensing mode or the grouping 
information (the same mark is given to image sensing 
data obtained in the same place) depending on the 
photographing place is given as appended information, 
the agreement between the given marks is the condition. 
Image sensing data meeting the grouping selection 
condition are sequentially supplied to the image 
sensing data holding unit 31 (S35 and S36) and held in 
it. 

[0068] 

A method of grouping image sensing data when a 
synthetic image is to be obtained by increasing the 
dynamic range as disclosed in Japanese Patent Laid-Open 
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No. 7-131796, for example, or a synthetic image is to 
be obtained by a method of panorama image synthesis for 
obtaining a wide-angle image from a plurality of image 
sensing data as disclosed in Japanese Patent Laid-Open 
No, 5-183789, for example, will be described below. In 
a case like this, it is desirable that appended 
information indicating that data is image sensing data 
for obtaining a certain synthetic image be given to the 
data during photography. If this is the case, in this 
embodiment, image sensing data constituting an image 
sensing data group are selected and grouped on the 
basis of the appended information. Alternatively, as 
explained with respect to the method of grouping the 
image sensing data using composition information, it is 
also possible to simply reproduce image sensing data, 
perform grouping determination, and group the image 
sensing data. In processing for obtaining synthetic 
image, all image sensing data stored in the image 
sensing data memory 20 are in some cases previously 
selected as image sensing data to be processed as a 
group when the processing is executed. If this is the 
case, in the present invention, the image reproduction 
is performed by processing all of the image sensing 
data stored in the image data memory 20 as a group of 
image sensing data. 
[0069] 
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As seen in the above -described embodiment, the 
gist of the present invention lies in that when image 
sensing data is to be converted into image data, a 
plurality of image sensing data meeting a predetermined 
5 condition are processed as one group of image sensing 
data, an image reproduction parameter is obtained from 
this image sensing data group, and the image sensing 
data belonging to the image sensing data group is 
converted into image data by using this image 

10 reproduction parameter. Therefore, various types of 
sequence processings can be considered without 
departing from the spirit and scope of the present 
invention as described above. These sequence 
processing operations are theorized or implemented in 

15 software, for example. Alternatively, these sequences 
are implemented in algorithm to the extent to which the 
implementation does not depart from the scope and 
spirit of the present invention, and a hardware 
apparatus operating in accordance with the algorithm 

20 can be applied. 
[0070] 

[Effects of the invention] 
As described above, according to the present 
invention, a plurality of image sensing data meeting a 
25 predetermined condition are processed as one image 

sensing data group and an image reproduction parameter 
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is determined on the basis of the image sensing data 
group* Then, the image sensing data belonging to the 
image sensing data group can be converted into image 
data by using the image reproduction parameter. 
Accordingly, the present invention solves the problem 
which occurred in the conventional image reproduction 
processing method, i.e., the problem that when an image 
reproduction parameter is determined from the image 
sensing data, a condition for determining the parameter 
is usually dissatisfied and hence the parameter cannot 
be accurately determined. Therefore, the image sensing 
data can be converted into image data using a more 
accurate image reproduction parameter, thus enabling 
accurate image reproduction. In addition, loss of the 
correlation between the image sensing data caused by 
the use of different image reproduction parameters for 
each of images (image sensing data) upon reproduction 
processing can be eliminated. The image sensing data 
can be converted into image data by using a more 
accurate image reproduction parameter in which the 
information and correlation between the respective 
images are held, thus enabling reproduction of natural 
and accurate images. 
[Brief Description of the Drawings] 
[Fig. 1] 

Fig. 1 is a block diagram showing an embodiment of an 
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image reproducing apparatus according to the present 
invention. 

[Fig. 2] 

Fig. 2 is a flow chart showing the operation of the 
5 embodiment of the present invention. 
[Fig. 3] 

Fig. 3 is a view for explaining an example of a 
photographed object in accordance with the embodiment 
of the present invention. 
10 [Fig. 4] 

Fig. 4 is a view for explaining an example of a 
photographed object in accordance with the embodiment 
of the present invention. 
[Fig. 5] 

15 Fig. 5 is a view for explaining an example of a 

photographed object in accordance with the embodiment 
of the present invention. 
[Fig. 6] 

Fig. 6 is a flow chart showing the operation of the 
20 embodiment of the present invention. 
[Fig. 7] 

Fig. 7 is a view for explaining an image sensing data 
selection condition given in the embodiment of the 
present invention. 
25 [Fig. 8] 

Fig. 8 is a view for explaining an image sensing data 
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selection condition range used in the embodiment of the 
present invention . 
[Fig. 9] 

Fig. 9 is a flow chart showing the operation of the 
5 embodiment of the present invention. 
[Fig. 10] 

Fig. 10 is a flow chart showing the operation of the 
embodiment of the present invention. 
[Description of the Reference Numerals] 



10 10 color image sensing unit (image sensing 

means) 

20 image sensing data memory (memory means) 

30 color image reproduction processor 

31 image sensing data holding unit 

15 32 image reproduction parameter determining unit 

(determining means) 

33 image sensing data selecting unit 

34 image sensing data selection designating 
unit (designating means) 

20 35 image reproduction processing unit 

36 control unit 

40 color image reproducing display 

50 reproduced image data memory 
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[Type of the Document] Abstract 
[Abstract] 

[Object] Desired accurate image reproduction can be 
performed . 

[Structure] An image reproducing apparatus comprises 
an image sensing data memory 20, an image sensing data 
selection designating unit 34, an image sensing data 
selecting unit 33 for selecting image sensing data from 
the image sensing data memory 20 , an image sensing data 
holding unit 31 for holding the image sensing data 
selected by the. image sensing data selecting unit 33, 
an image reproduction parameter determining unit 32 for 
determining an image reproduction parameter by using 
the selected and held image sensing data, and an image 
reproduction processing unit 35 for performing image 
reproduction of the image sensing data held by the 
image sensing data holding unit 31 by using the image 
reproduction parameter determined by the image 
reproduction parameter determining unit 32. 
[Selected Drawing] Fig. 1 
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